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(54) A method for automatically classifying images into events 



(57) A method tor automatically classifying images 
into events, the method includes the steps of: receiving 
a plurality of images having either or both date and/or 
time of image capture; determining one or more largest 
time differences of the plurality of images based on 
clustering of the images, arxi separating the plurality of 



Images into the events based on having one or more 
boundaries between events which one or more bounda- 
ries correspond to the one or more largest time differ- 
ences. 
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Description 

HELP OF THE INVENTION 

[0001] The invention relates generally to the field of s 
image processing having Image understanding that 
automatically classifies pictures by events and the like 
and, more particularly, to such automatic classification 
of pictures by time and date analysis and by block- 
based analysis which selectively compares blocks of io 
the images with each other. 

BACKGROUND OF THE INVENTION 

[0002] Pictorial images are often classified by the par- is 
ticular event, subject or the like for convenience of 
retrieving, reviewing, and albuming of the images. Typi- 
cally, this has been achieved by manually segmenting 
the images, or by the below-described automated 
method- The automated method includes grouping by 
color, shape or texture of the images for partitioning the 
images into groups of similar image characteristics. 
[0003] Although the presently known and utilized 
methods for partitioning images are satisfactory, there 
are drawbacks. The manual classification is obviously 
time consuming, and the automated process, although 
theoretically classifying the images into events, is sus- 
ceptible to miss-classification due to the inherent inac- 
curacies irrvolved with classification by color, shape or 
texture. 

[0004] Ck)nsequerrtly. a need exists for overcoming the 
above-described drawbacks. 

SUMMARY OF THE INVENTION 

[0005] The present invention is directed to overcoming 
one or more of the problems set forth above. Briefly 
summarized, according to one aspect of the present 
invention, the invention resides in a method for automat- 
ically classifying Images into events, the method com- 
prising the steps of: receiving a plurality of images 
having either or both date and/or time of image capture; 
determining one or more largest time differences of the 
plurality of Images based on clustering of or more larg- 
est time differences of the plurality of images based on 
clustering of the images; and separating the plurality of 
images into the events based on having one or more 
boundaries between events which one or more bourxla- 
ries correspond to the one or more largest time differ- 
ences. 

[0006] These and other aspects, objects, features and 
advantages of the present invention will be more clearly 
understood and appreciated from a review of the follow- 
ing detailed description of the preferred embodiments 
and appended claims, and by reference to the accom- 
panying drawings. 



ADVANTAGEOUS EFFECT OF THE INVENTION 

[0007] The present invention has the advantage of 
improved classification of images by utilizing both date 
and time information and block-based comparison that 
checks for similarity of subject and background in the 
images. If date and time information is not available, 
then the block-based analysis may be used as the sole 
basis for classification. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0008] 

Fig. 1 is a block diagram illustrating an overview of 
the present invention; 

Fig. 2 is a block diagram illustrating a date and time 

clustering technique of the present invention; 

Rg. 3 is a graph illustrating a scaling function used 

to map the result of the 2-means clustering; 

Fig. 4 is a graph illustrating a typical result of the 

scaling function of Fig. 3; 

Fig. 5 is a diagram illustrating a block diagram of an 
everrt boundary checking after the date artd time 
clustering; 

Rg. 6 is a diagram illustrating grouping of images 
witiiin each event based on content; 
Rg. 7 is a block diagram of a group-merging step of 
the present invention; 

Rg. 8 is a block diagram of image re-arrangement 
within each group; 

Rg. 9 Is a flowchart of block-t>ased histogram corre- 
lation technique; 

Rg. 10 is diagram Illustrating the conrparison 
between block histogram; 

Rg. 1 1 is diagram of an example of best intersec- 
tion mapping for three segment analysis; and. 
Rg. 12 is an illustration of shift detection within the 
t>lock t>ased histogram correlation. 

DETAILED DESCRIPTION OF THE INVENTION 

[0009] In the following description, the present inven- 
tion will be described in the preferred embodiment as a 
software program. Those skilled in the art will readily 
recognize that the equivalent of such software may also 
be constructed in hardware. 

[0010] Still further, as used herein, computer readable 
storage medium may comprise, for exarrple; magnetic 
storage media such as a magnetic disk (such as a 
floppy disk) or magnetic tape; optical storage rrvedia 
such as an optical disc, optical tape, or machine reada- 
ble bar code; solid state electronic storage devices such 
as random access memory (RAM), or read only mem- 
ory (ROM); or any other physical device or medium 
employed to store a conrputer program. 
[0011] In addition, the term "event" is defined herein 
as a significant occurrence or happening as perceived 
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by the subjective intent of the user of the image capture 
device. 

[C01 2] Before describing the present invention, it facil- 
itates understanding to note that the present invention is 
preferably utilized on any well-known computer system, 
such a personal computer. Consequently, the computer 
system will not be discussed in detail herein. It is also 
instructive to note that the images are either directly 
input into the computer system (for example by a digital 
camera) or digitized before input into the conputer sys- 
tem (for example by scanning), 

[G013] Referring to now Fig. 1, there is illustrated a 
flow diagram illustrating an overview of the present 
invention. Digitized images are input into the conrputer 
system where a software program of the present inven- 
tion will classify them into distinct categories. The 
images will first be ranked S10 in chronological order by 
analyzing the time and date of capture of each image. 
The date and/or time of capture of each picture may be 
extracted, for example, from the encoded information on 
the film strip of the Advanced Photo System (APS) 
images, or from information available from some digital 
cameras. The representations of the inr^ges will then be 
placed S20 into one of a plurality of distinct events by a 
date and time clustering analysis that is described 
below. Within each event, the contents of the in>ages 
are analyzed S20 for determining whether images clos- 
est in time to an adjacent event should be maintained in 
the event as defined by the clustering analysis, or the 
adjacent events merged together. After the images are 
defined into events, a further sub-classification (group- 
ing) within each event is performed. In this regard, the 
images within each event will then be analyzed by con- 
tent S30 for grouping images of similar content together, 
and then the date artd time S30 for further refining the 
grouping. 

[0014] The event segmentation 820 using the date 
and time information is by a k-means clustering tech- 
nique, as will be descrit)ed in detail t>elow, which groups 
the images into events or segments. A bourKlary check 
is then performed on the segments 820 for verifying that 
the tx)undary images should actually be grouped into 
the segment identified by the clustering, as will also be 
described below. 

[0015] These groups of images are then sent to a 
block-based histogram correlator S30 for analyzing the 
content- For each evertl or segment sent to the correla- 
tor, a content or subject grouping 830 is performed ther- 
eon for further sub-classifying the images by subject 
within the particular event segment. For example, within 
one event, several different subjects n^y appear, and 
these subject groupings define these particular sub- 
jects. The subject grouping is based primarily on image 
content, which is performed by a block-based histogram 
correlation technique. This correlation conrpares por- 
tions of two images with each other, as will also be 
described in detail below. The result of the ranking is the 
classification of images of each segment into distinct 



subject groupings. The date and time of all the images 
within each subject grouping are then compared to 
check whether any two or more sut^ject grouping can be 
merged into a single subject grouping 830. 
5 [001 6] A refinement and subject re-arrangement anal- 
ysis S40 will further improve the overall classification 
and the subject grouping by rearranging certain images 
within a subject group. 

[0017] Referring to Fig. 2. there is shown an exploded 
10 t3lock diagram illustrating the data and time clustering 
technique 820. First, the time interval between adjacent 
pictures (time difference) is computed S20a. A histo- 
gram of the time differences is then computed 820b, an 
example of which is shown in block 10. The abscissa of 
15 the histogram is preferably the time in minutes, and the 
ordinate of the histogram is the number of pictures hav- 
ing the specified time difference. The histogram is then 
mapped S20c to a scaled histogram using a time differ- 
ence scaling function, which is shown in Fig. 3. This 
20 mapping is to take the pictures with small time differ- 
ence and substantially maintain its time difference, and 
to take pictures with a large time difference and com- 
press their time difference. 

[0018] A 2-means clustering is then performed S20d 

25 on the mapped time-difference histogram for separating 
the maiDped histogram 1 0 into two dusters based on the 
time difference; the dashed line represents the separa- 
tion point for the two clusters. For further details of 2- 
means clustering. Introduction to Statistical Pattern 

30 Recognition, 2"^ edition by Keinosuke Fukunaga 1990 
may be consulted, and therefore, the process of 2- 
means clustering will not be discussed in detail herein. 
Referring briefly to Fig. 4. the result of 2-means cluster- 
ing is the segmentation of the histogram into two por- 

35 tions 10a and 10b. Normally, events are separated by 
large time differences. The 2-means clustering, there- 
fore, is to define where these large time differences 
actually exist. In this regard, the right portion 10b of the 
2-means clustering output defines the large time differ- 

40 ences that corresporKi to the event boundaries. 

[0019] Referring to Fig. 5. there is illustrated an exam- 
ple of boundary checking between everrts. For two con- 
secutive events i and j, a plurality of block-k»ased, 
histogram comparisons are made to check if the pic- 

45 tures at the border of one event are different from the 
pictures at the border of the other event. If the compari- 
son of conterrt is similar, the two segments are merged 
into one segment. Otherwise, the segments are not 
merged. Preferat>ly, the comparisons are performed on 

50 the three t>order images of each event (i3. i4. i5 with j1 , 
i2, j3), as illustrated in the drawing. For example, image 
i5 is compart with image j1 and etc. This block-based 
histogram comparison technique will be described in 
detail hereinbelow. 

55 [0020] Referring to Fig. 6. there is illustrated an over- 
view of subject (content) grouping for each segmented 
event. Within each segmented event i, adjacent pictures 
are compared (as illustrated by the an^ows) with each 
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other using the below-described, block-based histo- 
gram technique. For example, the block-based histo- 
gram technique may produce five subject groupings (tor 
example groups 1-5) from the one event I, as illustrated 
in the drawing. The arrangement of the subject grouping 5 
is stored for future retrieval during the subject an-ange- 
ment step &40- in particular, the subject grouping having 
a single image is staed (for example groups 2, 3, and 
5). 

[0021] Referring to Fig. 7. after the grouping by con- 7< 
tent, a bme and date ordering is performed on the 
grotptf>gs lor merging groups together based on a time 
and dale analysis A histogram of the time difference 
between adiaceni images in the event is computed, sim- 
ilar to Fig 4 A pi ©determined number of the largest 1 
time din«»t!nc«i (toi example boundary a^2 ) com- 
pared with fr^ txxmdanes (for example boundaries bi2. 
b23. I>i4. ) ol the subject grouping determined by the 
t>locit based analysis The predetermined nunrijer of 
largest t*T>e dittefeoces are determined by dividing the £ 
total njrrt>er dt images within an event by the average 
nurTt>er of picture per group (four is used in the present 
invention) H the boundary of the subject grouping 
matches the boundary based on the chosen time differ- 
ences, the subject groupr>QS will not be merged. If there i 
is not a n^tch i>etween the two boundaries, the subject 
grouptr>gs havmg a txxindary not having a matched 
time dif1ererx;e m the histogram will be merged into a 
single sut)ject groupwig (kx example groups bi, bb. bs 
merged into resulting group Ci). 

[0022] Refernng to Fig 8, there is illustrated a diagram 
of image re-arrangenr>ent Within each group. The 
arrangement of the initial subject groupings is retrieved 
for identifying subject groupings that contain single 
irr^ges (for example the groups with a single image of 
Fig. 6 -groups 2. 3. and 5 that are re-illustrated as 
groups 2, 3, and 5 in Fig 8). Any single images from the 
same subject grouping that are merged as identified by 
the merged subject grouping (for example, groups c^ 
and C2 of Fig. 7) are compared with all other images in 
the merged subject grouping, as illustrated by the 
arrows. This comparison is Ijased on block-based histo- 
gram analysis. If the comparisons are smilar. these 
images will re-arranged so that the similar images 
are located adjacent each other, for example groups d^ 
andd2- 

[0023] Further refinement is done by comparing any 
group that still contains a single image after the above 
procedure, with all the images in the event. This is to 
check if these single image groups can be better 
arranged within the event grouping. This comparison is 
similar to the subject re-arrangement step of Fig. 8. 
[0024] Referring to Fig. 9. there is illustrated a flow- 
chart of the block-based histogram correlation used in 
the atx>ve analyses. First, a histogram of the entire 
image of both images is computed S50, a global histo- 
gram. A comparison of the two histograms is performed 
by histogram intersection value S60 illustrated the fol- 



lowing equation: 

n 

£min(fli.C, 
Inter {R,C)= 

) where R is the histogram of the reference image. C is 
the histogram of the candidate image, and n is the 
number of bins in the histogram. If the intersection is 
under a threshold S65. preferably 0.34. although other 
thresholds may be used, the images are different. If the 
5 threshold Is met or exceeded S65. then a block-based 
histograrfi correlation will be performed S70. In this 
regard, each image will be divided into blocks of a given 
size, preferably 32 x 32 pixels in the present invention. It 
is instructive to note that those skilled in the art may 
0 vary the block size depending on the resolution of the 
image without departing from the scope of the invention. 
For each block, a color histogram is computed. Refer- 
ring to Fig. 10, if one image is considered a reference 
image and one image a candidate image, the images 
5 are compared in the following way. Each block 20 of the 
reference image is compared to the corresponding 
block 30 of the candidate image and to the adjacent 
blocks 40, 8 blocks in the present invention. 
[0025] Referring to Fig. 9, the block histograms 
0 between the reference image and the candidate image 
are compared using tiie histogram intersection equation 
defined above S80. The average intersection value is 
derived by computing the average of the best intersec- 
tion values from each of the block comparisons S90. 
is This average intersection value will be compared to a 
low threshold (preferably 0.355). and a high threshold 
(preferably 0.557). If the average intersection value is 
below the low threshold S95, the two images are con- 
sidered different If tiie average intersection value is 
to above the high threshold S96, then tine two images are 
considered similar. If the average intersection value is 
between these two thresholds, further analysis will be 
performed as described below (3-segment average 
intersection nrap S100). 
45 [0026] Referring to both Hgs. 9 and 11 . a 3-segment 
analysis will be performed to determine if the tvw> 
images may contain a similar subject. This is performed 
by first forming a map 60 which contains tiie average of 
the two highest intersection values of each of the block 
so comparisons; for example. 9 comparisons were per- 
formed in the illustration of Fig. 10. the average of tiie 
highest two will be used for map 60. Fig. 1 1 illustrates, 
for example, a 9 x 6 block although it should be under- 
stood tiiat the map size depends on ttie size of tiie 
55 image. This map is divided into three parts: the left por- 
tion 70a. the center portion 70b, and tiie right portion 
70c. If the average intersection value of the center por- 
tion 70b Is higher than a threshold (preferafcrfy 0.38) 
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S1 05, the two images may contain a very similar subject 
in the center portion 70b of the image, and the two 
images may fc)e considered to be similar by subject. In 
addition, the comparisons of the histogram will be per- 
formed with the referertce and candidate images 
reversed. If the two images are similar both methods 
should give substantially similar conrelation; obviously if 
they are different, the resufts will not be similar. The 
images are then checked S1 10 to determine if there is a 
high intersection value in one of the directions, right, 
left, up, and down. 

[0027] Referring to Figs. 9 and 12. shift detection is 
used to determine the case when the two images 90 
and 1 00 (of two different sizes in the drawing) have very 
similar subject that appears in different locations of the 
image. For example, the main subject may be situated 
in the center of one image and to the left-hand side of 
the other image. Such a shift can be determined by 
recording both the best intersection values of the refer- 
ence blocks, as well as the coordinates of the corre- 
sponding candidate blocks. This is achieved by 
comparing the intersection values of the blocks in four 
directions (right, left, up, and down). The entire image 
will be shifted by one block (as illustrated by the arrows) 
in one of the directions (right in the drawing) where the 
best intersection value is the highest. The above analy- 
sis and the shift can be repeated S120to check for sim- 
ilarity. 

[0028] The invention has been described with refer- 
ence to a preferred embodiment. However, it will be 
appreciated that variations and modifications can be 
effected by a person of ordinary skill in the art without 
departing from the scope of the invention. 



performing the 2-means clustering. 

4. The method as in claim 2. wherein step (c) includes 
checking the images adjacent the one or more 

5 boundaries for similarity by comparing content of 

the images. 

5. The method as in claim 4, wherein step (c) includes 
checking the images adjacent the one or more 

10 kx)ur7daries for similarity by using a block-based his- 
togram correlation technque. 

6. The method as in claim 5 further comprising step 
(d) dividing the events into subject grouping by 

15 using an image content analysis. 

7. The method as in claim 6, wherein step (d) includes 
dividing the events into subject grouping by using a 
tDlock-based histogram technique. 

20 

8. A method for automatically classifying images into 
events, the method conprising the steps of: 

(a) receiving a plurality of images arranged in 
25 chronological order; 

(b) dividing the images into a plurality of blocks; 
and. 

(c) grouping the images into subject grouping 
based on block-based histogram correlation 

30 which includes computing a color histogram of 

each block and computing a histogram inter- 
section value which determines the similarity 
between blocks. 



Claims 

1. A method for automatically classifying images into 
events, the method comprising the steps of: 

(a) receiving a plurality of images having either 
or both date and/or time of image capture; 

(b) determining one or more largest time differ- 
ences of the plurality of images based on clus- 
tering of the images; and, 

(c) separating the plurality of images into the 
events based on having one or more bourida- 
ries between events which one or more bound- 
aries correspond to the one or more largest 
time differences. 

2. The method as in claim 1 , wherein step (b) includes 
computing a time difference histogram and per- 
forming a 2-means clustering on the time difference 
histogram for defining the one or more boundaries. 

3. The method as in claim 2. wherein step (b) further 
includes mapping the time difference histogram 
through a time difference scaling function before 



35 9. The method as in claim 8. wherein step (c) includes 
comparisons of two of the images by shifting one of 
tiie images in a desired direction biased on the inter- 
section value and then computing tiie block based 
conrelation. 

40 

10. The method as in claim 9. wherein step (c) includes 
forming a nnap that contains two best intersection 
values of each of the block comparisons; dividing 
tiie map into three portions; and then conparing 
45 center portions for similarity. 
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